nlp-speech

Speech-to-Text Conversion

Techniques, Tools, and Trade-offs

Omkar Ninav
June 30, 2025

Good [morning/afternoon], thank you for taking the time to attend this presentation.

The goal today is to give a structured and technical overview of speech-to-text conversion — how it works, what tools and models are currently leading the field, and what trade-offs are involved in choosing a particular solution.

This is a technology that’s become foundational in a wide range of domains — from customer support and assistive tech, to real-time captioning and voice interfaces. I’ll be focusing on how Python-based tools like Whisper, Wav2Vec2, and others can be used to build robust STT systems.

By the end, I hope to outline clearly: - How the core pipeline functions, - Which models are best suited for various use cases, and - What the cost and accuracy implications are — especially in terms of practical deployment.

Let’s begin with the basics.

What is Speech-to-Text?

Converts spoken language into written text
Also known as Automatic Speech Recognition (ASR)
Used in applications like:
- Virtual assistants (e.g., Siri, Alexa)
- Captioning & subtitling
- Meeting transcription
- Voice commands

How Speech-to-Text Works

Audio Input
Voice signal captured from a microphone or file
Feature Extraction
Converts raw audio into numerical features (e.g., MFCCs, spectrograms)
Acoustic Model
Maps features to phonemes or characters using ML/DL models
Language Model
Predicts word sequences for context-aware transcription
Decoder
Aligns acoustic and language models to produce final text

flowchart LR
  A[Audio Input<br/>Voice signal] --> B[Feature Extraction<br/>MFCCs / Spectrograms]
  B --> C[Acoustic Model<br/>ML/DL-based]
  C --> D[Language Model<br/>Contextual Prediction]
  D --> E[Decoder<br/>Final Text Output]

This slide breaks down the standard architecture behind most modern speech-to-text systems.

It starts with the audio input, typically a microphone recording or audio file, which is passed to a feature extraction module. This module transforms the raw waveform into meaningful patterns — usually through methods like MFCCs (Mel-Frequency Cepstral Coefficients) or spectrograms.

Next, the acoustic model interprets these features and maps them to likely phonemes or character sequences. Traditionally, these were HMM-based, but today they’re often deep neural networks like CNNs, RNNs, or transformers.

The language model plays a critical role — it uses linguistic context to predict the most likely word sequences. This is what helps the system understand the difference between similar-sounding phrases like “ice cream” vs “I scream.”

Finally, the decoder combines all of this — the acoustic predictions and the language probabilities — to produce the final transcription.

Understanding this pipeline helps clarify where different models and libraries fit in, and what we can tweak based on accuracy, speed, or deployment constraints.

Techniques: Traditional ASR

HMMs + GMMs
Early models for mapping acoustic features to phonemes
n-gram Language Models
Predict word sequences based on prior word probabilities
Feature Extraction (MFCCs)
Converts raw audio into spectral features

Let’s start with traditional ASR systems.

Early speech recognition was built using Hidden Markov Models (HMMs) and Gaussian Mixture Models (GMMs). HMMs model the sequential nature of speech, and GMMs handle the probability distributions of acoustic features within each state.

For predicting the next word or phrase, systems used n-gram language models — for example, a trigram model would look at the two previous words to estimate the next one. These models were easy to implement but had limited ability to understand context or long-term dependencies.

The front end of these systems used MFCCs (Mel-Frequency Cepstral Coefficients) to convert audio waveforms into feature vectors that captured the shape and tone of the speech signal. MFCCs remain popular even in some deep learning systems due to their compact and informative representation of sound.

While these traditional methods laid the foundation for modern ASR, they struggled with complex variations like accents, overlapping speakers, and noisy environments.

Techniques: Deep Learning & Modern Trends

RNNs, CNNs, Transformers
Deep models for sequence modeling and attention
End-to-End Models (CTC / Attention)
Learn transcription directly from audio input
Trends
- Self-supervised learning (e.g. Wav2Vec 2.0)
- Multilingual support (e.g. Whisper)
- On-device inference (edge devices, mobile)

Modern speech recognition systems have shifted toward deep learning.

Initially, Recurrent Neural Networks (RNNs) were used because they could model sequences — perfect for speech. Then came Convolutional Neural Networks (CNNs) to handle time-frequency patterns in audio. Today, transformers are dominant due to their superior performance in modeling long-range dependencies and context.

Instead of breaking the process into multiple modules like traditional ASR, deep learning systems often use end-to-end models. These can be trained to map directly from raw audio to text using techniques like Connectionist Temporal Classification (CTC) or attention-based models.

Recent trends in STT are pushing the field even further: - Self-supervised learning enables models like Wav2Vec 2.0 to learn from raw, unlabeled audio — drastically reducing the need for costly transcriptions. - Models like Whisper offer multilingual support and are robust to accents and background noise. - There’s also growing interest in on-device inference, where models are small and efficient enough to run on phones, edge devices, or embedded systems — ensuring both privacy and speed.

These modern techniques make STT more accurate, scalable, and accessible than ever before.

Python Libraries for Speech-to-Text (1/2)

Whisper (OpenAI)
- Transformer-based, multilingual
- High accuracy, runs offline
SpeechRecognition
- Simple API wrapper for Google, IBM, etc.
- Easy for beginners
Wav2Vec 2.0 (Hugging Face)
- Pretrained self-supervised models
- High-quality transcriptions

Python Libraries for Speech-to-Text (2/2)

DeepSpeech (Mozilla)
- Lightweight and fast
- Less accurate on noisy inputs
Kaldi (via PyKaldi)
- Research-grade toolkit
- Steeper learning curve

Model Comparison: Features at a Glance

Model	Accuracy	Offline	Multilingual	Ease of Use	Cost
Whisper	✅✅✅	✅	✅✅✅	✅✅	Free
Wav2Vec 2.0	✅✅	✅	⚠️ (mostly English)	✅	Free
Google API	✅✅✅	❌	✅✅✅	✅✅✅	Paid
DeepSpeech	✅	✅	❌	✅✅	Free
Kaldi	✅✅	✅	✅ (with effort)	⚠️ Complex	Free

This table summarizes the strengths and limitations of some of the most widely used speech-to-text models and toolkits.

Starting with Whisper, it stands out as a balanced option — it offers high accuracy, works offline, supports multiple languages, and is open-source. It does require a decent GPU to run efficiently, especially for long audio files or real-time use, but it’s very accessible for developers and teams with modest resources.

Wav2Vec 2.0 is another excellent option. It provides high accuracy, especially when fine-tuned on domain-specific data. However, it’s primarily trained on English datasets, so multilingual support is more limited unless you retrain it. It’s open-source and can be accessed via Hugging Face.

Google Speech-to-Text API is a commercial solution. It offers very high accuracy and supports over 100 languages. It’s excellent for real-time use cases and scales easily. However, it’s not free — the cost is around $1.44 per hour, and it requires internet connectivity, which may raise privacy concerns in sensitive environments.

DeepSpeech was designed to be lightweight and fast, making it suitable for edge devices. That said, it has started to lag behind in terms of accuracy, especially on noisy or accented inputs, and its development has slowed down in recent years.

Kaldi is extremely powerful and flexible. It’s used in many academic and enterprise applications where low-level control is needed — for instance, custom language models, speaker adaptation, and complex pipelines. However, it’s not beginner-friendly, and the learning curve is steep. PyKaldi can help if you want to work with it in Python.

This slide helps us decide which model might be most appropriate depending on our project constraints — whether we prioritize cost, offline access, multilingual support, or ease of use.

In the next slide, we’ll break down the cost aspect more specifically, including free tiers and usage pricing for cloud APIs.

Cost: Open-Source Models

Model	Cost	Offline Use	Cloud Required
Whisper	Free (Open Source)	✅	❌
Wav2Vec 2.0	Free (Open Source)	✅	❌
DeepSpeech	Free (Open Source)	✅	❌
Kaldi	Free (Open Source)	✅	❌

Cost: Cloud APIs

Service	Approx. Cost	Offline Use	Cloud Required
Google Speech API	~$1.44 per hour	❌	✅
AWS Transcribe	~$1.44 per hour	❌	✅
Azure Speech	~$1.60 per hour	❌	✅

Cost – Summary & Recommendation

Open-Source Models (Whisper, Wav2Vec, etc.)
- ✅ Free and offline
- ⚠️ Require setup and local resources
Cloud APIs (Google, AWS, Azure)
- ✅ Easy to use, scalable
- ⚠️ Ongoing cost and privacy trade-offs

✔️ Recommendation:
Use Whisper or Wav2Vec 2.0 for local, cost-effective transcription
Use Cloud APIs only for real-time or highly multilingual needs

Conclusion & Summary

STT is a mature, versatile technology
Open-source tools (Whisper, Wav2Vec) offer high quality with no cost
Cloud APIs provide convenience but incur recurring costs
Model choice depends on:
✅ Accuracy
✅ Cost
✅ Deployment constraints
✅ Privacy needs

✔️ Recommended:
Use Whisper for secure, offline transcription
Use cloud APIs only where real-time & scalability are critical

Thank You!

Questions? Suggestions?

🔗 View this presentation on GitHub:

Presented by Omkar Ninav — June 2025